Imaging device modulating intensity of light with grating pattern

ABSTRACT

An image sensor has a plurality of pixels arranged in an array on an imaging surface and converts a captured optical image into an image signal. A modulator is provided on a light receiving surface of the image sensor and modulates the light intensity using a grating pattern. An image processor performs image processing of the image signal output from the image sensor. The modulator has a grating substrate and a first grating pattern is formed on a first side of the grating substrate adjacent to a light receiving side of the image sensor. The grating pattern is composed of a plurality of concentric circle patterns. Each concentric circle pattern has concentric circles having a pitch which becomes smaller in inverse proportion to the distance from the center thereof. The plurality of concentric circle patterns do not overlap with each other in the grating pattern.

TECHNICAL FIELD

The present invention relates to an imaging device, and in particular toa technique for miniaturizing an imaging device.

BACKGROUND ART

A camera installed in a smartphone and an in-vehicle camera for 360°sensing, needs miniaturization. According to US2014/0253781A (PTL 1), itis written that a special diffraction grating substrate is attached toan image sensor which obtains the image of the outside world byobtaining the incident angle of the incident light by inverse problemcalculation from the projection pattern generated by the light passingthrough the diffraction grating substrate on the sensor without a lens.

According to US2015/0219808A (PTL 2), it is described that a concentricgrating pattern having a pitch which becomes smaller from the centertoward the outside is used as the diffraction grating substrate.

PTL 1: U.S. Patent Application Publication No. 2014/0253781 A

PTL 2: U.S. Patent Application Publication No. 2015/0219808 A

SUMMARY OF INVENTION Technical Problem

In PTL 1 described above, the problem is that the pattern of thediffraction grating to be formed on the upper side of the substratewhich is attached to the image sensor is a special grating pattern suchas a spiral shape and the calculation for solving the inverse problemfor reproducing the image becomes complicated from the projectionpattern received by the sensor.

In PTL 1 described above, since the diffraction grating in which theconcentric circular grating patterns overlap each other is used, thereis a concern that the transmissivity decreases and mutual concentriccircular grating patterns interfere with each other, leading to anincrease in noise of the reproduced image.

An object of the present invention is to provide an easy technique fordetecting the angle of incidence of light and to provide an imagingdevice which reduces a reduction in light utilization efficiency at thattime and does not generate interference noise of mutually concentriccircular grating patterns.

Solution to Problem

One of the representative imaging apparatuses of the present inventionfor solving the above the problem, for example, A single concentricgrating pattern in which the pitch becomes smaller inverselyproportional to the distance from the center is formed on the objectside of the substrate to be attached to the sensor and the lighttransmitted through it is again modulated by another concentric gratingpattern in which the pitch becomes smaller inversely proportional to thedistance from the center, and an image of the external object isobtained from the two-dimensional Fourier transform image of themodulated image.

According to the present invention, an external object image can beobtained by a simple operation such as Fast Fourier Transform (FFT).Further, it is possible to reduce the decrease in transmittance of lightand to eliminate the interference noise of the mutually overlappingconcentric circular grating patterns remaining in the reproduced image.Since it does not use a lens, it is also effective for maintenance freevehicle cameras and surveillance cameras that are concerned aboutdeterioration of image quality due to aged deterioration andmisalignment of lenses. The problems, configurations, and effects otherthan those described above will be clarified by the description of theembodiments below.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram illustrating a basic example of the presentinvention.

FIG. 2 is a diagram showing a state of taking an image of an objectoutside according to a basic embodiment of the present invention.

FIG. 3 is a diagram for explaining that the projection of an image fromthe grating substrate's front side to the back side by obliquelyincident parallel light causes in-plane shift.

FIG. 4 is schematic diagram for explaining the generation of moirefringes and frequency spectrum when the grating axes of both sides ofthe grating substrate are aligned.

FIG. 5 is a schematic diagram in the case where the axes of the frontside grating and the back side grating are arranged to be shifted fromeach other.

FIG. 6 is schematic diagram for explaining generation of moiré fringesand frequency spectrum in the case where grating of both sides of thegrating substrate are shifted and arranged

FIG. 7 is diagram showing calculation results of spatial frequencyspectral image when irradiated with 10 lights in total includingvertical incidence plane wave and plane waves with 9 different incidentangles

FIG. 8 is a diagram showing the calculation result of the spatialfrequency spectral image when irradiated with a total of 10 lightsincluding a vertical incident plane wave and plane waves with 9different incident angles.

FIG. 9 is a diagram for explaining the angle that light from each pointconstituting an object makes with the sensor.

FIG. 10 is a diagram showing the spatial frequency spectrum in the casewhere two grating patterns are shifted in the horizontal direction.

FIG. 11 is a diagram showing the spatial frequency spectrum in the casewhere two grating patterns are shifted in the vertical direction.

FIG. 12 is a diagram showing an embodiment in the case where the backside grating pattern is replaced with the sensor sensitivitydistribution.

FIG. 13 is a diagram showing that when the object to be imaged is at afinite distance, the projection of the front side grating pattern ontothe back side is enlarged from the front side grating pattern.

FIG. 14 is a diagram showing an embodiment in which the surface gratingpattern is displayed with variable size by a liquid crystal element.

FIG. 15 is diagram showing an embodiment in the case where the frontside grating is replaced with a cylindrical lens array

FIG. 16 is a diagram showing an embodiment of a smartphone on which animaging device of the present invention is mounted

FIG. 17 is a block diagram of the image processing circuit of embodiment1.

FIG. 18 is a block diagram of the image processing circuit of embodiment3.

FIG. 19 is a block diagram of the image processing circuit of embodiment4.

FIG. 20 is an illustration showing a grating pattern of an area of 3×3division of embodiment 7.

FIG. 21 is a diagram showing the arrangement of initial phases of bothsides of a grating pattern of an area of 3×3 division in embodiment 7.

FIG. 22 is a diagram showing an image of a moiré fringe that isgenerated when light from a single point light source enters the 3×3divided double-sided grating of embodiment 7.

FIG. 23 is a diagram showing an image of moiré fringes obtained by noisereduction image processing according to embodiment 7.

FIG. 24 is the image processing circuit block diagram of embodiment 7.

FIG. 25 is a diagram showing a grating pattern of an area of 2×2division of embodiment 8.

FIG. 26 is a diagram showing an image of moiré fringes obtained byperforming noise reduction image processing according to embodiment 8.

FIG. 27 is block diagram of the image processing circuit of Embodiment8.

FIG. 28 is a diagram showing an embodiment of a vehicle in which theimage device of the present invention is mounted as an in-vehiclecamera.

DESCRIPTION OF EMBODIMENTS

Embodiments are explained with respect to the figures.

Embodiment 1

FIG. 1 is a basic configuration diagram of the present invention. Theimaging device 101 includes a double-sided grating substrate 102, animage sensor 103, and an image processing circuit 106. The double-sidedgrating substrate 102 is fixed in close contact with the light receivingside of the image sensor 103, and on the front side of the double-sidedgrating substrate 102, concentric circles having a grating interval(pitch) narrow in inverse proportion to the radius from the centertoward the outside shaped grating pattern 104 is formed. A similargrating pattern 105 is also formed on the back side of the image sensor103 which is in contact with the light receiving side. The intensity oflight passing through these grating patterns is modulated by the gratingpattern. The transmitted light is received by the image sensor 103, andthe image signal is subjected to image processing by the imageprocessing circuit 106 and output to the monitor display 107 or thelike. In a usual imaging device, a lens for forming an image is requiredin front of the sensor, but in the present invention, it is possible tomake an image of an external object without a lens. In this case, theconcentric circular grating pattern 104 has no other intersectinggrating pattern in each ring pattern constituting the concentric circle,so no unnecessary interference between the grating patterns occurs, andit is possible to suppress a decrease in light utilization efficiency itcan.

FIG. 2 shows an illustration of capturing the subject 201 by the imagingdevice of FIG. 1 and outputting it to the monitor display 107. And thegrating plane of the double-sided grating substrate 102 faces thesubject 201 so as to photograph it.

FIG. 17 is a block diagram showing the processing contents of the imageprocessing circuit 106. For the Moire fringe image to be input, atwo-dimensional FFT operation is performed for each color RGB componentto obtain a frequency spectrum. Data of one side frequency is cut outand an intensity calculation is performed. Further, noise removal,contrast emphasis processing and the like are performed on the obtainedimage, the color balance is adjusted, and output as a photographedimage.

The imaging principle will be described below. First, a concentric gridpattern in which the pitch becomes smaller in inverse proportion to theradius from the center outward is defined as follows. Let us assume acase of a spherical wave, close to a plane wave, and a plane wave, usedas a reference light, interfere with each other in a laserinterferometer or the like. When the radius from the referencecoordinate of the center of the concentric circle is r and the phase ofthe spherical wave is φ (r), this can be expressed as Equation 1 byusing the coefficient β which determines the magnitude of the curvatureof the wavefront.ϕ(r)=βr ²  (1)

As for the spherical wave, it is expressed by the square of the radius rand the spherical wave is close to a plane wave, so it can beapproximated using only the lowest order of expansion. When plane wavesare caused to interfere with light having this phase distribution, theintensity distribution of the interference fringes as shown in Equation2 is obtained.I(r)=¼|exp iϕ(r)+1|²=½(1+cos ϕ)  (2)=½(1+cos βr ²)

This is a concentric stripe with a bright line at the radial positionsatisfying Equation 3.ϕ(r)=βr ²=2nπ(n=0,1,2,Λ)  (3)

Assuming that the pitch of the stripe is p, Equation 4 is obtained.

$\begin{matrix}{{{p\frac{d}{dr}{\phi(r)}} = {{2\; p\;\beta\; r} = {2\pi}}}{{p(r)} = \frac{\pi}{\beta\; r}}} & (4)\end{matrix}$

It turns out that the pitch narrows in inverse proportion to the radius.Such stripes are called Fresnel zone plates.

A grating pattern having a transmittance distribution proportional tothe intensity distribution is used as the lattice patterns 104 and 105shown in FIG. 1.

Assuming that parallel light enters the substrate having the thickness twith such a grating formed on both sides at an angle θ₀ as shown in FIG.3, light whose geometrical optics is multiplied by the transmittance ofthe grating of the front side with the refraction angle in the substrateas θ is incident on the back side with a deviation of δ=t·tan θ. And,assuming that the centers of the two concentric circle gratings arealigned, the transmittance of the grating of the back side is multipliedby shifting by δ. In this case, intensity distribution expressed byEquation 5 is obtained.

$\begin{matrix}\begin{matrix}{{{I\left( {x,y} \right)}{I\left( {{x + \delta},y} \right)}} = {\frac{1}{4}\left\{ {1 + {\cos\;{\beta\left( {x^{2} + y^{2}} \right)}}} \right\}\left\{ {1 + {\cos\;{\beta\left( {\left( {x + \delta} \right)^{2} + y^{2}} \right)}}} \right\}}} \\{= {\frac{1}{8}\left\{ {2 + {4\;\cos\;{\beta\left( {r^{2} + {\delta\; x}} \right)}\cos\;{\delta\beta}\; x} +} \right.}} \\\left. {{\cos\; 2{\beta\left( {r^{2} + {\delta\; x}} \right)}} + {\cos\; 2{\beta\delta}\; x}} \right\}\end{matrix} & (5)\end{matrix}$

It can be seen that the fourth term of this expansion formula createsstraight striped patterns at equal intervals in one overlapping area inthe direction of the shift of the two gratings. Stripes that occur atrelatively low spatial frequencies due to the superimposition of suchstripes and fringes are called moire fringes. Such straight stripes atequal intervals produce a sharp peak in the spatial frequencydistribution obtained by the two-dimensional Fourier transform of thedetected image. It is possible to obtain the value of δ, that is, theincident angle θ of the light beam from the value of the frequency. Itis obvious that such moire fringes uniformly spaced on the whole surfaceoccur at the same pitch regardless of the direction of deviation becauseof the symmetry of the concentric grid arrangement.

It is thought that it is impossible to obtain uniform fringes on thewhole surface with other grating patterns, because such a stripe isobtained by forming the grating pattern with the Fresnel zone plate.Even in the second term in the equation, it can be seen that fringesmodulated by Moiré fringes are generated in the Fresnel zone plate, butsince the frequency spectrum of the product of the two fringes is aconvolution of each Fourier spectrum, a sharp peak is not obtained. Fromthe Equation 5, only the component having a sharp peak is extracted asshown in Equation 6.I(x,y)=⅛(2+cos 2δβx)  (6)

And the Fourier spectrum is as shown in Equation 7;F[I(x,y)]=⅛F[2+cos 2δβx]=¼δ(u,v)+⅛δ(u+δβ/π,v)+⅛δ(u−δβ/π,v)  (7)

Here, F represents the calculation of the Fourier transform, u and v arethe spatial frequency coordinates in the x and y directions, and δ withthe parentheses is the delta function. From this result, it can be seenthat in the spatial frequency spectrum of the detected image, the peakof the spatial frequency of moire fringes occurs at the position ofu=±δβ/π.

This state is shown in FIG. 4. The figure shows the layout of the rayand the substrate from the left, the moiré fringes, the schematicdiagram of the spatial frequency spectrum, where the top row is (a)vertical incidence, the middle row is (b) when the ray is incident atangle θ from the left side, and the bottom row is (c) rays are incidentat angle θ from the right side. The front side grating pattern 104 andthe back side grating pattern 105 on the grating substrate 102 arealigned with each other. In (a), since the shadow of the front sidegrating pattern 104 and the shadow of the back side grating patterncoincide with each other, moire fringes do not occur. In (b) and (c),the same moire fringes occur because the deviation between the frontside grating pattern 104 and the back side grating pattern is equal, thepeak position of the spatial frequency spectrum also coincides, and itcannot be determined from the spatial frequency spectrum whether theincident angle of the light beam is (b) or (c).

In order to avoid this, as shown in FIG. 5, the two gratings are shiftedrelative to the optical axis in advance so that the shadows of the twolattices overlap with each other even for light rays incidentperpendicularly to the substrate It is necessary to keep it. When therelative shift of the shadows of the two lattices with respect to thevertical incidence plane wave on the axis is δ₀, the deviation δ causedby the plane wave of the incident angle θ can be expressed as Equation8.δ=δ₀ +t tan θ  (8)

At this time, the peak of the spatial frequency spectrum of moiréfringes of the ray of incidence angle θ is at the position of Equation 9on the plus side of the frequency.

$\begin{matrix}{u = {\frac{\delta\beta}{\pi} = {\frac{1}{\pi}\left( {\delta_{0} + {t\;\tan\;\theta}} \right)\beta}}} & (9)\end{matrix}$

Assuming that the size of the image sensor is S, the number of pixels inthe x, y directions are both N, the spatial frequency spectrum of thediscrete image by fast Fourier transform (FFT) changes from −N/(2S) to+N/(2S), the position is obtained in the range. Considering that theincident angle on the plus side and the incidence angle on the minusside are uniformly received from this, it is reasonable the spectralpeak position of the moiré fringe due to the vertical incidence planewave (θ=0) is the same as the origin (DC) position, for example, and thecenter position with respect to the frequency position at the end, thatis, the spatial frequency position is expressed by Equation 10.

$\begin{matrix}{{\frac{1}{\pi}\delta_{0}\beta} = \frac{N}{4\; S}} & (10)\end{matrix}$

Therefore, it is reasonable to set the relative center offset of the twogratings.

$\begin{matrix}{\delta_{0} = \frac{\pi\; N}{4\beta\; S}} & (11)\end{matrix}$

Thus, it is reasonable to set the relative center offset of the twogratings to Equation 11.

FIG. 6 shows the arrangement of the light beam and the substrate, andthe schematic diagram of the moire fringe and its spatial frequencyspectrum when the front side grating pattern 104 and the back sidegrating pattern 105 are shifted in advance. In the same way as in FIG.4, the arrangement of FIG. 6 shows the light beam and the substrate onthe left side, the moiré stripe in the center row, and the spatialfrequency spectrum on the right side. The upper row (a) shows when theray is perpendicularly incident, the middle row (b) shows when lightrays are incident from the left side at an angle θ, and the lower rowshows the case (c) when light rays are incident at angle θ from theright side. Since the front side grating pattern 104 and the back sidegrating pattern 105 are arranged before being shifted by δ₀ in advance,moiré fringes also occur in (a), and a peak appears in the spatialfrequency spectrum. As described above, the shift amount δ₀ is set sothat the peak position appears at the center of the spectral range onone side from the origin. At this time, since the deviation δ becomeslarger in (b) and becomes smaller in (c), the difference between (b) and(c) can be distinguished from the peak position of the spectrum unlikein FIG. 4. The spectral image of this peak is a bright point indicatinga light flux at infinity, which is nothing but a captured image by theimaging device of the present invention.

Assuming that the maximum angle of incidence of parallel light that canbe received is θ max;

$\begin{matrix}{u_{\max} = {{\frac{1}{\pi}\left( {\delta_{0} + {t\;\tan\;\theta_{\max}}} \right)\beta} = \frac{N}{2\; S}}} & (12)\end{matrix}$

From Equation 12, the maximum field angle that can be received by theimaging device of the present invention is given by Equation 13.

$\begin{matrix}{{\tan\;\theta_{\max}} = \frac{\pi\; N}{4\; t\;\beta\; S}} & (13)\end{matrix}$

From the analogy with imaging using ordinary lenses, it is consideredthat parallel light of the angle of view θ_(max) is received at the endof the sensor by focusing.

$\begin{matrix}{f_{eff} = {\frac{S}{2\;\tan\;\theta_{\max}} = \frac{2\;\beta\;{tS}^{2}}{\pi\; N}}} & (14)\end{matrix}$

The effective focal length of the imaging optical device of the presentinvention which does not use a lens is expressed by Equation 14,respectively.

As shown by equation 2, the transmittance distribution of the lattice isbasically supposed to have a sinusoidal characteristic, but if such acomponent is present as the fundamental frequency component of thegrating, it is conceivable the transmittance of the grating is binarizedto change the duty of the grating region having high transmittance andthe duty of low region, and increase the width of the high transmittanceregion to increase the transmittance.

In the above explanation, in both cases, only one incident angle issimultaneously incident light. However, in order for the presentinvention to actually function as a camera, we should assume that lightsof a plurality of incident angles are incident at the same time. Suchlights having a plurality of incident angles overlap the images of theplurality of front side grating pattern 104 at the time when they areincident on the back side grating pattern. If these mutually generatemoiré fringes, there is a concern that it will become noise whichhinders the detection of moiré fringes with the back side gratingpattern which is a signal component. In practice, however, thesuperimposition of the images of the front side lattice 104 does notcause a peak of the moiré image, and only the overlap with the back sidegrating pattern will produce a peak. The reason will be explained below.First, it is a big difference that the overlapping of the shadows of thefront side lattice 104 by the rays of a plurality of incident angles isa sum, not a product. With the overlap of the shadow of the front sidegrating pattern 104 due to the light of one incidence angle and the backside grating pattern 105, the light intensity distribution after passingthrough the back side grating pattern 105 is obtained by multiplying theintensity distribution of the light which is the shadow of the frontside by the transmittance of the back side grating pattern 105. On theother hand, since the overlapping of the shadows caused by the lightshaving different angles incident on the front-side grating pattern 104is the overlapping of the lights, it is not a product but a sum.I(x,y)+I(x+δ,y)=½{1+cos β(x ² +y ²)}+½{1+cos β((x+δ)² +y ²)}=1+cos β(r ² +δx)cos δβx  (15)

In the case of a sum, it becomes a distribution obtained by multiplyingthe distribution of the lattice of the original Fresnel zone plate bythe distribution of the moiré fringes as shown by Equation 15.Therefore, since its frequency spectrum is represented by the overlapintegral of each frequency spectrum, even if the spectrum of Moiré alonehas a sharp peak, actually only the ghost of the frequency spectrum ofthe Fresnel zone plate will occur at that position. That is, a sharppeak does not occur in the spectrum. Therefore, even if light having aplurality of incident angles is entered, the spectrum of the moiré imagedetected is always only the moire of the product of the front sidegrating pattern 104 and the back side grating pattern 105, and as longas the back side grating pattern 105 is single, the peak is only a onefor one incident angle. In order to confirm the principle, the resultsof the simulation performed are shown in FIGS. 7 and 8.

All of them are in the spectrum, on the condition that the sensor sizeis 20 mm, the viewing angle θ max=±70°, the incident side and exit sidelattice coefficient β=50 (rad/mm 2), δ₀=0.8 mm, the number of pixels is1024×1024, the substrate thickness is 1 mm, the ratio is 1.5, on thecondition that incident light with θx=50° and θy=30° and incident lightwith θx=−30° and θy=70° and θx=10°, θy=−20° incident light, incidentlight with θx=20°, θy=30°, incident light with θx=30°, incident lightwith θy=−40°, incident light with θx=−10°, θy=40°, Incident light atθx=−20°, θy=−30°, incident light at θx=−30°, θy=0° and incident lightwith θx=40° and θy=50° in total or 10 plural waves are incident. FIG. 7is a black and white inverted image of the spectral image, and FIG. 8 isa diagram of the luminance of the spectral image. The original moiréimage itself was also omitted because the grating pitch is also smalland cannot be visually recognized even if it is displayed as the drawingof this specification. In the figure, the whole area of the spatialfrequency spectral region with the center at the DC component and theperiphery at ±N/2S is displayed. Since the DC component has a largevalue, it is removed by masking and only the peak component to bedetected is displayed. Furthermore, since the peak width of the spectrumis narrow as it is, it is difficult to visually recognize, so thecontrast is emphasized. Further, in FIG. 7, the position of the signalpeak is displayed surrounded by circles. In the diagram of FIG. 8, sincethe drawn line cannot be displayed without passing through the peak asit is, the result of applying the averaging filter of the mesh size isdisplayed. In each case, basically, it is shown that 10 peaks can bedetected as a total of 20 peaks on the positive and negative sidesacross the origin. In the present embodiment, the pitch of the outermostcircumference of the grating pattern was about 6 μm and the effectivefocal length was 12.4 mm.

Here, the correspondence between the parallel light that has beenexplained so far and the light from the actual object will be describedschematically with reference to FIG. 9. Strictly speaking, the lightfrom each point constituting the subject 201 is incident as a sphericalwave from a point light source onto the integrated grating sensorsubstrate 901 of the imaging apparatus of the present invention. At thistime, when the Integrated grating sensor substrate 901 is sufficientlysmall or sufficiently far from the object, it can be considered that theincident angle of light illuminating the integrated grating sensorsubstrate 901 from each point is the same. From the relationship thatthe spatial frequency displacement Δu of the moire fringe with respectto the minute angular displacement Δθ obtained from the equation (9) is1/S or less, which is the minimum resolution of the spatial frequency ofthe sensor, the condition under which Δθ can be regarded as parallellight can be expressed as Equation 16.

$\begin{matrix}{{{\Delta\; u} = {{\frac{1}{\pi}\beta\; t\;{\Delta\theta}} \leq \frac{1}{S}}}{{\Delta\theta} \leq \frac{\pi}{S\;\beta\; t}}} & (16)\end{matrix}$

From this, for example, in the condition of the present embodimentΔθ<0.18°, which is a condition that is possible if the sensor size of 20mm is 6 m away from the object.

From the analogy of the above results, it can be seen that the imagingdevice of the present invention is capable of imaging on an object atinfinity.

Embodiment 2

In the present embodiment, the case where the output image islandscape-long will be described. In the above-described embodiment, asshown in FIG. 10, the front side grating pattern 104 and the back sidegrating pattern 105 are shifted to the left and right. At this time, ifthe shape of the sensor is a square, and the pixel pitch is also thesame in the x direction and the y direction, as shown on the right sideof the figure, the spatial frequency spectrum of the sensor output isreproduced with the image being separated left and right within thefrequency range of both x and y±N/S. However, at this point, the imageis basically limited to a vertically long area. Generally, the imageacquired by the camera is a landscape rectangle. Therefore, it isdesirable to arrange it as shown in FIG. 11 as an arrangement suitablefor it. At this time, the front side grating pattern 104 and the backside grating pattern 105 are vertically shifted, and the images formedin the spatial frequency space of the sensor output are separated upwardand downward. In this way, the output image can be made horizontallylong.

Embodiment 3

In the present embodiment, a case where a moiré fringe is virtuallygenerated in a processed image will be described. In the above-describedembodiment, the same grating pattern is arranged on the front side andthe back side of the grating substrate so as to be shifted from eachother, the angle of the incident parallel light is detected from thespatial frequency spectrum of the moire fringe to form an image. Thedifference between the present embodiment is that there is no back sidegrating pattern 105 in terms of the configuration and that the role ofthe back side grating pattern 105 is performed by the image processingunit. Since the grating pattern on the back side, closely attaching tothe sensor, is an optical element that modulates the intensity ofincident light by setting the sensitivity of the sensor effectively bytaking account of the transmittance of the grating on the back side,Moire fringes can be caused virtually in the inside. FIG. 12 shows anembodiment in the case where the back side grating pattern 105 is notprovided on the back side of the grating substrate. By doing so, onegrating to be formed can be reduced, so that the manufacturing cost ofthe device can be reduced. However, at this time, it is necessary thatthe pixel pitch of the sensor is small enough to sufficiently reproducethe grating pitch or the grating pitch is rough enough to sufficientlyreproduce at the pixel pitch of the sensor. In the case of forming thelattice on both sides of the substrate, it is not always necessary toresolve the pitch of the grating by the pixels of the sensor, since itis sufficient that only the moiré image can be resolved, and it ispossible to determine the grating pitch independently of the sensorpitch. However, when reproducing a lattice with a sensor, the resolutionof the lattice and the resolution of the sensor must be equal. As shownin FIG. 18, in the image processing circuit 1201, since processing ofback side grating intensity modulation corresponding to a back sidegrating pattern for generating a moiré image to the sensor output isadded, an intensity modulation circuit for performing this processing isalso necessary.

In this way, if the grating can be made variable, that is, the size ofthe concentric circles can be made variable, the detection light is notnecessarily parallel light. As shown in FIG. 13, when a spherical wavefrom a point 1301 constituting an object illuminates the front sidegrating pattern 104 and its shadow 1302 is projected on the bottom, theimage projected on the bottom is expanded almost evenly. Therefore, bydirectly multiplying the transmittance distribution of the bottomgrating designed for the parallel light, a linear moire fringe at equalintervals does not occur. However, if the grating of the bottom side isenlarged in accordance with the shadow of the uniformly enlarged topside grating, linear moire fringes at equal intervals can be generatedagain for the enlarged shadow 1302. Thus, it is possible to selectivelyreproduce the light from the object point 1301 which is not necessarilyat infinity. Therefore, focusing becomes possible.

Embodiment 4

In this embodiment, a liquid crystal element is used for the grating ofthe front side substrate. As in the third embodiment, the grating of thefront side substrate can also be made variable by using a liquid crystalelement or the like. FIG. 14 is a diagram showing a state in which thevariable grating 1403 on the front substrate is displayed by the liquidcrystal element sealed with the liquid crystal substrate 1402 with theliquid crystal layer 1401 interposed therebetween. Transparentelectrodes are formed on the liquid crystal substrate 1402 and thegrating substrate 102, and an arbitrary lattice image can be displayedvia electrodes (not shown). Light from an object point 1301 which isbasically closer to infinity and has a finite distance is divergentlight, then in order for the shadow of the front side grating pattern tobe the same size on the back side grating pattern at the back side ofgrating substrate, it is only necessary to display it slightly shrinkedon the front side. As shown in FIG. 19, a liquid crystal driving circuitis built in the image processing circuit 1404, and a front side gratingpattern is generated according to an external focus specification input,and a front side grating pattern corresponding to an arbitrary focusposition is displayed on the liquid crystal element. By using the liquidcrystal element for the grating of the front side substrate in thismanner, the grating of the front side substrate can be varied, that is,the size of the concentric circle can be made variable.

Embodiment 5

In this embodiment, as shown in FIG. 15, each grating of the incidentside grating is a cylindrical lens 1501. All of the grating lines areformed in an array shape as cylindrical lenses. Thereby, it is possibleto reduce the light amount loss by the shielding portion of thegray-scale grating and to improve the light utilization efficiency ofthe imaging device of the present invention.

Embodiment 6

In the present embodiment, as shown in FIG. 16, an embodiment in thecase where the imaging device of the present invention is mounted on asmartphone is shown. Since the aperture can be enlarged while keepingthe imaging device thin, the effective focal length can be increasedaccording to the equation 14, so that the aperture becomes small and thefocal length becomes short as in the conventional smartphone camera. Itis possible to solve the problem that the image is not blurred. A focusadjustment knob 1602 is attached to the object side, and it is possibleto capture an image of an object at an arbitrary distance by making itvariable according to focus designation for inputting a back sidegrating pattern to be displayed on a liquid crystal element incorporatedas a unit with the sensor.

Embodiment 7

In the explanation of the principle of the first embodiment, it isexplained that the sharp peak of the signal is obtained only in thefrequency of the moire fringe of the fourth term in the Equation 5, butdepending on the conditions of the optical system and the subject, Thesecond and third terms become noise, which may affect the image qualityof the reproduced image. Here, a configuration for removing these noiseswill be described.

A block diagram of the process flow in the present embodiment is shownin FIG. 24, and in addition to FIG. 17, a process of removing Moirefringe noise is added.

In order to remove noise, in the present embodiment, as shown in FIG.20, the front side grating pattern 104 and the back side grating pattern105 are formed in independent zone plates in an area divided into 3×3 ormore. However, each zone plate is arranged without overlapping. At thistime, in the front side grating pattern 104, the initial phases of thegratings are aligned to φ 1, φ 2, and φ 3, respectively, on the leftside, the middle side, and the right side in the upper stage, the middlestage, the lower stage, and the backside grid 105. As shown in FIG. 21,these are arranged so that the phase of the front side grating pattern104 and the phase of the back side grating pattern are overlapped witheach other independently of each other in all combinations. At thistime, the light intensity I_(s) on the side of the sensor can beexpressed by the Equation 17

$\begin{matrix}\begin{matrix}{{I_{S}\left( {x,y,\phi_{F},\phi_{B}} \right)} = {\left\{ {\sum\limits_{k}\;{I_{k}\left( {x,y,\phi_{F}} \right)}} \right\}{I\left( {{x + \delta_{0}},y,\phi_{B}} \right)}}} \\{= {\frac{1}{4}\left\lbrack {\sum\limits_{k}\;{A_{k}\left\{ {1 + {\cos\left\{ {{\beta\left( {\left( {x + \delta_{k}} \right)^{2} + y^{2}} \right)} + \phi_{F}} \right\}}} \right\}}} \right\rbrack}} \\{\left\{ {1 + {\cos\left\{ {{\beta\left( {\left( {x + \delta_{0}} \right)^{2} + y^{2}} \right)} + \phi_{B}} \right\}}} \right\}} \\{= {\frac{1}{8}{\sum\limits_{k}\;{A_{k}\left\{ {2 + {2\;\cos\left\{ {{\beta\left( {r^{2} + {2\; x\;\delta_{k}}} \right)} + \phi_{F}} \right\}} +} \right.}}}} \\{{2\;\cos\left\{ {{\beta\left( {r^{2} + {2\; x\;\delta_{0}}} \right)} + \phi_{B}} \right\}} +} \\{{\cos\left\{ {{2{\beta\left( {r^{2} + {\left( {\delta_{k} + \delta_{0}} \right)x}} \right)}} + \phi_{F} + \phi_{B}} \right\}} +} \\\left. {\cos\left\{ {{2{\beta\left( {\delta_{k} - \delta_{0}} \right)}x} + \phi_{F} - \phi_{B}} \right\}} \right\}\end{matrix} & (17)\end{matrix}$

Ik is the light intensity distribution of the shadow of the front sidegrating pattern 104 by the kth point light source, and I is thetransmittance distribution of the back side grating pattern 105. Theinitial phase φF of the front side grating pattern 104 and the initialphase φ B of the back side grating pattern 105 each take three values ofφ 1, φ 2, φ 3 as described above. It is assumed that the kth point lightsource illuminating the front side grating pattern 104 emits light withthe intensity of Ak and forms the shadow of the front side gratingpattern 104 on the sensor with a deviation of δk. The second term in { }in the lower part of Equation 17 is the shadow of the front sidegrating, the third term is the intensity modulation of the back sidegrating, the fourth term is the sum of frequency components of the twogratings, the fifth term is the difference frequency and also a term ofMoire fringes which is a signal component of the present invention.Therefore, it suffices to extract only the component having the addedphase of φF−φB.

Expression (17) is expressed as Expression 18 by focusing on φF and φB;

$\begin{matrix}{{I_{S}\left( {\phi_{F},\phi_{B}} \right)} = {\frac{1}{8}{\sum\limits_{k}\;{A_{k}\begin{Bmatrix}{2 + {2\;{\cos\left( {\theta_{1\; k} + \phi_{F}} \right)}} + {2\;{\cos\left( {\theta_{2} + \phi_{B}} \right)}} +} \\{{\cos\left( {\theta_{3\; k} + \phi_{F} + \phi_{B}} \right)} + {\cos\left( {\theta_{4\; k} + \phi_{F} - \phi_{B}} \right)}}\end{Bmatrix}}}}} & (18)\end{matrix}$By using the orthogonality of trigonometric functions, coefficients ofcos φ B and sin φ B can be extracted as shown in Equations 19 and 20.

$\begin{matrix}{{\frac{1}{8}{\sum\limits_{k}\;{A_{k}\left\{ {{2\;\cos\;\theta_{2}} + {\cos\left( {\theta_{3\; k} + \phi_{F}} \right)} + {\cos\left( {\theta_{4\; k} + \phi_{F}} \right)}} \right\}}}} = {\frac{1}{\pi}{\int_{0}^{2\pi}{{I_{S}\left( {\phi_{F},\phi_{B}} \right)}\cos\;\phi_{B}d\;\phi_{B}}}}} & (19) \\{{\frac{1}{8}{\sum\limits_{k}\;{A_{k}\left\{ {{{- 2}\;\sin\;\theta_{2}} - {\sin\left( {\theta_{3\; k} + \phi_{F}} \right)} + {\sin\left( {\theta_{4\; k} + \phi_{F}} \right)}} \right\}}}} = {\frac{1}{\pi}{\int_{0}^{2\pi}{{I_{S}\left( {\phi_{F},\phi_{B}} \right)}\sin\;\phi_{B}d\;\phi_{B}}}}} & (20)\end{matrix}$

From this, it is possible to express as follows by extracting the termsof cos ϕ B cos ϕ F, sin ϕ B sin ϕ F by Equations 21 and 22.

$\begin{matrix}{{\frac{1}{8}{\sum\limits_{k}\;{A_{k}\left\{ {{\cos\;\theta_{3\; k}} + {\cos\;\theta_{4\; k}}} \right)}}} = {\frac{1}{\pi^{2}}{\int_{0}^{2\pi}{\int_{0}^{2\pi}{{I_{S}\left( {\phi_{F},\phi_{B}} \right)}\cos\;\phi_{B}\cos\;\phi_{F}d\;\phi_{B}d\;\phi_{F}}}}}} & (21) \\{{\frac{1}{8}{\sum\limits_{k}\;{A_{k}\left\{ {{{- \cos}\;\theta_{3\; k}} + {\cos\;\theta_{4\; k}}} \right)}}} = {\frac{1}{\pi^{2}}{\int_{0}^{2\pi}{\int_{0}^{2\pi}{{I_{S}\left( {\phi_{F},\phi_{B}} \right)}\sin\;\phi_{B}\sin\;\phi_{F}d\;\phi_{B}d\;\phi_{F}}}}}} & (22)\end{matrix}$

Add these side by side, Equation 23 is obtained

$\begin{matrix}{{\frac{1}{8}{\sum\limits_{k}\;{A_{k}\cos\;\theta_{4\; k}}}} = {\frac{1}{2\pi^{2}}{\int_{0}^{2\pi}{\int_{0}^{2\pi}{{I_{S}\left( {\phi_{F},\phi_{B}} \right)}\left( {{\cos\;\phi_{B}\cos\;\phi_{F}} + {\sin\;\phi_{B}\sin\;\phi_{F}}} \right)d\;\phi_{B}d\;\phi_{F}}}}}} & (23)\end{matrix}$

This eventually corresponds to extracting only the moire component inequation 17 as shown in equation 24.

$\begin{matrix}{{\frac{1}{8}{\sum\limits_{k}\;{A_{k}\cos\left\{ {2{\beta\left( {\delta_{k} - \delta_{0}} \right)}x} \right\}}}} = {\frac{1}{2\pi^{2}}{\int_{0}^{2\pi}{\int_{0}^{2\pi}{{I_{S}\left( {\phi_{F},\phi_{B}} \right)}{\cos\left( {\phi_{B} - \;\phi_{F}} \right)}d\;\phi_{B}d\;\phi_{F}}}}}} & (24)\end{matrix}$

This operation corresponds to scanning and integrating both the phase ofthe front side grating pattern 104 and the phase of the back sidegrating pattern 105 in two dimensions. In order to discretize thisintegration at three minimum sampling points, a 3×3 double-sided Fresnelzone plate described in FIGS. 20 and 21 is used. FIG. 22 shows Moiréfringes obtained by this 3×3 double-sided grating with a single pointlight source. A plurality of Fresnel zone plates appear to intersect,but this is a light intensity distribution formed in a sensor shape bylight incident on the Fresnel zone plate on both sides withoutintersection. FIG. 23 shows a moiré fringe obtained by synthesizing themoiré fringes from each 3×3 cell by using the equation (24). Unnecessarynoise other than moire fringes is greatly reduced. As a result, thequality of the captured image can be improved.

Embodiment 8

FIG. 25 shows another example of the front side grating pattern 104. Ablock diagram of the processing flow in the present embodiment is shownin FIG. 27, and a process of removing Moire fringe noise is added incomparison with FIG. 18. This is based on the assumption that the backside grating pattern 105 is not a fixed grating, but a sensorsensitivity is virtually given to the Fresnel zone plate shape or aliquid crystal element (or the like) is used. The front side gratingpattern is divided into 2×2 areas, and the initial phase of each Fresnelzone plate is shifted by 90°. In the seventh embodiment, the phases areshifted in columns or rows, but in this embodiment the phase isdifferent in all the regions. FIG. 26 shows a regenerated image of moiréfringes by a single point light source. Noise is further reduced ascompared with FIG. 23. In this way, by making different the phase of thevirtual zone plate on the back side by 4 steps in each phase by 90° inall regions and generating moiré fringes, it is possible to perform theintegration operation in 4 phases on both side gratings. Further, as thedivision becomes smaller, further improvement of the noise reductioneffect can be expected.

Embodiment 9

FIG. 28 shows an embodiment in which the imaging device of the presentinvention is utilized for sensing 360° of the automobile 2801. Since theimaging device 101 of the present invention does not use a lens, it isexpected to be thin, and strong against deviation and deterioration overtime, and low in price. Therefore, it expected that many fields of view2802 can be arranged so as to cover the entire area around theautomobile, eliminating the blind spot of the driver, and contributingto the realization of a safe and secure society.

The present invention is not limited to the examples described above,but includes various modified examples. For example, the examplesdescribed above have been described in detail in order to explain thepresent invention in an easy-to-understand manner, and are notnecessarily limited to those having all the configurations described.

Further, a portion of the configuration of a certain example can bereplaced by the configuration of another example, and the configurationof another example can be added to the configuration of a certainexample.

Further, it is possible to add, delete, and replace other configurationsfor a portion of the configuration of each example.

Each of the configurations, functions, processing units, processingmeans, and the like described above may be realized in hardware bydesigning a portion or all of them, for example, with an integratedcircuit. Each of configurations, functions, and the like described abovemay be realized by software by allowing a processor to interpret andexecute a program which realizes each function. Information such as aprogram, a table, a file or the like that realizes each function can bestored in a recording device such as a memory, a hard disk, a solidstate drive (SSD), or a recording medium such as an IC card, an SD card,or a DVD.

Also, control lines and information lines indicate what is considered tobe necessary for explanation, and all control lines and informationlines are not necessarily indicated for products. In practice, it can beconsidered that almost all the configurations are mutually connected.

REFERENCE SIGNS LIST

-   -   101: imaging device    -   102: double-sided grating substrate    -   103: image sensor    -   104: front side grating pattern    -   105: back side grating pattern    -   106, 1201: image processing circuit    -   107: monitor display    -   201: subject    -   901: Integrated grating sensor substrate    -   1301: object point    -   1302: shadow of front grating pattern    -   1401: liquid crystal layer    -   1402: liquid crystal substrate    -   1403: variable grating    -   1501: cylindrical lens    -   1601: smartphone    -   1602: focus adjustment knob    -   2801: automobile

The invention claimed is:
 1. An imaging device comprising: a modulatorfor modulating an intensity of light using a first grating pattern, animage sensor for converting the light transmitted through the modulatorinto an electric signal to generate a sensor image; and an imageprocessing unit that applies image processing to the signal output fromthe image sensor, wherein the first grating pattern comprises aplurality of concentric circular patterns, wherein the concentriccircular pattern is composed of a plurality of concentric circles whosepitch becomes smaller in inverse proportion to the distance from centerreference coordinates of the concentric circular pattern, wherein theplurality of concentric circular patterns do not overlap with each otherin the first grating pattern, wherein the image processing unit performsa two-dimensional Fourier transform operation on the signal output fromthe image sensor, wherein the modulator has a second grating pattern,and wherein the modulator intensity-modulates light transmitted throughthe first grating pattern with the second grating pattern and outputsthe light to the image sensor.
 2. The imaging device according to claim1, wherein the second grating pattern is disposed on a surface so as toface a surface on which the first grating pattern is formed.
 3. Theimage device according to claim 1, wherein the image processing unitincludes an intensity modulation unit that performs processing forvirtually intensity modulating light transmitted through the firstgrating pattern, wherein the modulator outputs the light transmittedthrough the first grating pattern to the image sensor, and the imagesensor outputs the captured image to the image processing unit, andwherein the intensity modulating unit performs a process of intensitymodulating the light transmitted through the first grating pattern withrespect to an image captured from the image sensor using a virtualsecond grating pattern.
 4. The imaging device according to claim 3,wherein the intensity modulation unit intensity-modulates lighttransmitted through the first grating pattern by changing the size ofthe concentric circle of the virtual second grating pattern.
 5. Theimaging device according to claim 1, wherein the reference coordinateposition of the first grating pattern and the reference coordinateposition of the second grating pattern are deviated in mutually oppositedirections with respect to an axis passing through the center of theimage sensor and perpendicular to the light receiving side.
 6. Theimaging device according to claim 5, wherein the reference coordinateposition of the first grating pattern and the reference coordinateposition of the second grating pattern are shifted in the short sidedirection of the image output from the image processing unit.
 7. Theimaging device according to claim 1, wherein the first grating patternis formed by a cylindrical lens.
 8. The imaging device according toclaim 1, wherein the first grating pattern is divided into a pluralityof regions, and different concentric circular patterns are arranged foreach region.
 9. The imaging device according to claim 8, wherein phasesof the concentric circular patterns are independent for the each region.10. The imaging device according to claim 1, wherein the first gratingpattern and the second grating pattern are divided into a plurality ofregions, and the concentric circular patterns which are differentbetween adjacent regions are arranged, and a combination of a phase of agrating of the plurality of regions in the first grating pattern and aphase of the plurality of regions in the second grating pattern arearranged so as to be overlapped by a plurality of combinations.
 11. Theimaging device according to claim 10, wherein the image processing unitperforms an operation of extracting moire fringe components of theoutput image, and, the calculation is performed based on a combinationof the phase of the plurality of regions in the first grating patternand the phase of the plurality of regions in the second grating pattern.12. An imaging device comprising: a modulator for modulating anintensity of light using a first grating pattern, an image sensor forconverting the light transmitted through the modulator into an electricsignal to generate a sensor image, and an image processing unit thatapplies image processing to the signal output from the image sensor,wherein the modulator has a liquid crystal element displaying the firstlattice pattern, wherein the first grating pattern comprises a pluralityof concentric circular patterns, wherein the concentric circular patternis composed of a plurality of concentric circles whose pitch becomessmaller in inverse proportion to the distance from center referencecoordinates of the concentric circular pattern, and wherein the liquidcrystal element can change the size of the concentric circles of thefirst grating pattern.